Skip to content

Refactor store-backing calculator function#708

Merged
yousefmoazzam merged 5 commits intomainfrom
refactor-determine-store-backing
Apr 22, 2026
Merged

Refactor store-backing calculator function#708
yousefmoazzam merged 5 commits intomainfrom
refactor-determine-store-backing

Conversation

@yousefmoazzam
Copy link
Copy Markdown
Collaborator

@yousefmoazzam yousefmoazzam commented Apr 8, 2026

The main changes are:

  • correcting mistakes in what chunk sizes should be accounted for when determining the backing of the dataset store for a given section
  • simplifying the top-level determine_store_backing() function

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have made corresponding changes to the documentation
  • I have added the user-release-note label in order to include this PR in the "Notable
    Changes for Users" section in release notes

@yousefmoazzam yousefmoazzam force-pushed the refactor-determine-store-backing branch from f5a9001 to ac1e96e Compare April 8, 2026 13:30
The padded shape of the next section's input chunk was being used, which
is incorrect from the perspective of the order of operations that occur
when setting up a section's source and sink in `_setup_source_sink()` in
the task runner. Instead, it should be the padded shape of the current
section's input chunk.
…culation

This is purely for the sake of making the order of the chunk size
calculations in `determine_store_backing()` match the order of the
allocations that would occur during the setup + processing of a section.
For the last section:
- the input chunk's backing (RAM or hdf5 file) is purely determined by
  the output of the previous section
- the output chunk is not written anywhere (the dummy sink simply
  discards the blocks given to it)

Furthermore, the dummy sink doesn't take a type of backing as a
parameter (and if it did, it's unclear what it could do with that
information, given that the dummy sink simply discards the input data).

Therefore, there's no need to have any calculation of the backing of the
store for the last section's writing.
Due to the previous commit, there is now only one dataset store backing
calculator function that needs a reduction operation performed, so there
is no need to have a generic way to produce a decorator that can perform
the reduction on the result of different functions.
@yousefmoazzam yousefmoazzam force-pushed the refactor-determine-store-backing branch from ac1e96e to 78873e6 Compare April 22, 2026 09:38
@yousefmoazzam yousefmoazzam marked this pull request as ready for review April 22, 2026 09:42
@yousefmoazzam yousefmoazzam merged commit 899a43d into main Apr 22, 2026
5 of 6 checks passed
@yousefmoazzam yousefmoazzam deleted the refactor-determine-store-backing branch April 22, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant